Categorization according to Language : A step toward combiningLinguistic Knowledge and Statistic Learning

In this article, we address the problem of categorization according to language by presenting a method based on natural properties of language which allow us to categorize any kind of sentence with a very high success rate. The major diiculties in categorization are convergence and textual errors. Convergence since dealing with short entries involve discarding languages from few clues. Textual errors since documents coming from diierent electronic ways may contain spelling and grammatical errors as well as character recognition errors generated by OCR. We describe an approach to sentence cate-gorization based on natural properties of languages. We combine several linguistic knowledge sources and we study their role. While our rst system achieved a high success only on long sentences, we studied how overcoming the weaknesses and we present an interesting use of statistical pro-les. Tested for french, english, spanish and german discrimination, the improvements are appreciable, achieving in one test a perfect discrimination for texts, a perfect discrimination for sentences of more than 7 words and a very high discrimination rate for very short sentences. The resolution power is based on the comprehension that frequent phenomena are diierent depending on whether processing long inputs or processing short inputs. Thus, several kinds of linguistic knowledges are necessary and a way to combine them is presented. Having linguistic features of each language at its disposal, the system computes for each of them its likelihood to be selected. The name of the language having the optimum likelihood tag the input | but non resolved ambiguities are maintained. The implementation is fast, small, robust, evolutive and textual errors tolerant.